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ABSTRACT 

Many studies suggest factors that might influence online learning and assessment, but most have not been empirically 
tested. We use survey data from 91 students to investigate what factors influence students’ satisfaction with online 
assessment, and overall student satisfaction with the learning management system - Moodle. The survey questionnaire 
was sent anonymously to all students who took the online or hybrid course sections at a private Midwest university in 
summer 2013, as well as to all instructors who taught those courses. The result showed that overall student satisfaction 
with online learning is significantly affected by how the course is organized and how the content is sequenced, the ease 
with which students can complete assignments, and the use of the system to engage students with content. 


KEYWORDS 

Online Learning, Assessment Of Learning, Student Satisfaction, Learning Management Systems, Moodle Introduction 


1. INTRODUCTION 

According to Allen and Seaman (Allen & Seaman, 2013), the Sloan Online Survey found that in 2002 less 
than half of U.S. higher education institutions reported online education as being critical to their long-term 
strategy. However, in 2012, 70% of the surveyed academic leaders saw online learning (or e-learning) as 
critical to their long-term strategy. The 2012 report showed that there were 6.7 million students (32% of the 
total student population in the U.S.) taking at least one online course. Furthermore, 77% of academic leaders 
rated the learning outcomes in online education as the same or superior to face-to-face instruction. Thus, on- 
line learning is here to stay and we have to learn to use it effectively. Higher education continues to evolve, 
including recent increases in the number of courses offered fully and partially (hybrid) online. In order to 
offer these courses, it is important to have an effective technology platform to support the online class 
environment and activities. Many new technologies are used to support online education, particularly 
learning management systems (LMS's), which serve as the core technology platforms for the online 
environment. Some popular systems include Blackboard, Moodle, Sakai, Desire2Learn, Canvas, and 
eCollege. Moodle as an open technology platform is emerging as the most cost effective solution. Hence, 
our study uses Moodle and we believe our findings can be applicable to all LMSs. 

LMS systems support many important functions for effective online education, including facilitating 
instruction, assessment, and course administration as well as providing a new means for communication with 
students. To create a strong foundation for successful online education, it is critical for universities to 
measure how well the systems they use deliver on the key functions and support the online learning 
environment to serve their core academic mission. The use of new technologies and the application of new 
educational models need to be supported by systematic redesign of the processes at both the institutional and 
educator levels (Georgouli, Skalkidis, & Guerreiro, 2008). Hence, there is a need to understand factors that 
affect student satisfaction with online education and the effectiveness of the platform as an effective learning 
and communicating tool. 
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2. LITERATURE REVIEW 

Online learning is rapidly growing in society today; however, there are many variables that help explain the 
contrasts between online and traditional face-to-face learning as well as how they influence student 
satisfaction. According to Sun et al. (Sun, Tsai, Finger, Chen, & Yeh, 2008), some of the factors that 
influence student satisfaction with online learning systems include: computer/technology anxiety on the part 
of the learner, instructor attitudes toward e-learning, course flexibility and quality, perceived usefulness of 
the LMS and ease of use, and the diversity in assessment methods. Student satisfaction with online learning 
is a significant predictor of learning outcomes, and could be influenced by course structure, instructor 
feedback, students’ self-motivation and learning style, interaction and instructor facilitation (Eom, Wen, & 
Ashill, 2006). According to Bell and Federman (Bell & Federman, 2013), online learning should provide 
content, immersion, interactivity, and effective communication. A comparison study conducted by Summers, 
Waigandt and Whittaker of student achievement and satisfaction in an online versus face-to-face statistics 
course found that although students taking the online course learned as much as students in the face-to-face 
course, online course students were less satisfied with the method of delivery as compared to students take 
the course face-to-face (Summers, Waigandt, & Whittaker, 2005). By focusing on the factors that affect 
student satisfaction, LMSs can be improved and implementation strengthened to increase the learner 
satisfaction and retention of the material. 

Online learning presents an array of challenges and issues to instructors who depend on technology 
systems to completely deliver and support their courses. There are concerns about the validity, practicality, 
and reliability of online student assessment (Dermo, 2009), as well as concerns about the security (Alwi & 
Fan, 2010) and academic honesty (King, Guyette, & Piotrowski, 2009). Stacey and Wisenberg noted the 
importance of key variables, such as length of time teaching face-to-face and online, overall teaching load, 
class size, and institutional context that affect faculty members’ motivation and attitudes towards moving 
from traditional to virtual classrooms (Stacey & Wiesenberg, 2008). 

Naveh et al. (Naveh, Pliskin, & Tubin, 2010) found significant correlation between LMS use and student 
satisfaction in terms of similar organizational variables, which included class size, course content, instructor 
status, and the existence of forums. Given that a properly implemented and supported online learning system 
can help students' and instructors' initial online experiences run more smoothly and ease the transitions from 
face-to-face classrooms to virtual learning, it is important to study the factors that influence successful 
adoption of online learning. A LMS is one of the most representative e-learning applications. The LMS is 
used for online education as well as to supplement face-to-face courses. It is commonly used to post a 
course’s syllabus and announcements, homework assignments and projects, and lecture notes and slides for 
students to access online (ODCD, 2005). There is a strong movement toward open-source solutions (away 
from proprietary software) for e-learning applications (Coppola & Neelley, 2004). Open-source software 
provides flexibility to combine languages, scripts, learning objects, and lesson plans without the steep cost of 
proprietary packages (Williams, 2003). Moodle is an example of such an open-source LMS, and is the 
platform that is the subject of our study. 

According to Georgouli et al. (Georgouli et al., 2008), online content should be complemented and 
enhanced through activities (e.g., blog, discussion board, etc.) to facilitate self-learning. Additionally, 
according to Selim (Selim, 2005), instructors of online courses need to have a positive attitude towards the 
technology, and the students should have competent computer skills. Consequently, one of the biggest 
challenges with online learning is that students and instructors may lack the knowledge or skills required to 
use an online LMS for the first time. A poor first experience can scare students away from online learning in 
the future (Georgouli et al., 2008). 

Regardless of whether it is delivered online or face-to-face, effective teaching involves providing an 
opportunity for student-faculty interaction, active learning, and prompt feedback. The research findings 
suggested that online education can be a superior mode of instruction with timely, meaningful instructor 
feedback of various types (Eom et al., 2006). Yet, in the online environment, interactions, learning, and 
feedback often require unique strategies due to the challenges presented by technology-mediated teaching. 
Students take online courses are expected to take greater control of their learning process and be more active 
in stimulating their peers’ learning, therefore facilitation of online learning emerges as an important role in 
guiding these student-centered approaches (Baran, Correia, & Thompson, 2011). 
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Moreover, as the hierarchy in the online environment is flattened with more distributed power and control 
(Schmm & Hong, 2002), instructors are expected to adopt more facilitative approaches in creating learner- 
centered online classrooms (Salmon, 2004; Smith, 2005). While there is still a strong focus on the 
responsibilities of instructors in online courses, the instructor moves from being at the center of the 
interaction or the source of information to the "guide on the side," which implies that instructors design, 
organize, and schedule the activities and learners assume greater responsibility for their learning by 
coordinating and regulating their learning activities (Anderson, Rourke, Garrison, & Archer, 2001; Berge, 
2009) (p. 429). 

The notion of teaching online requires the development of new skills and sets of pedagogies has led 
researchers to study the roles that online instructors take in online education environments (Anderson et al., 
2001; Berge & Collins, 2000; Goodyear, Salmon, Spector, Steeples, & Tickner, 2001; Graham, Cagiltay, 
Lim, Craner, & Duffy, 2001; Guasch, Alvarez, & Espasa, 2010; Salmon, 2004). When designing online 
courses, it is important to consider the format and effectiveness of student assessment. According to Reeves 
(Reeves, 2000), there are different approaches to incorporate alternative assessments into online education, 
such as cognitive, performance, and portfolio assessments. The use of assessment rubrics for student 
assessments would make assessment activities more reliable and valid to provide more formal measures of 
achievement (Oncu & Cakir, 2011). 

2.1 Hypotheses 

Although some studies suggest factors that might influence online learning and assessment, they were not 
focused on the influence of LMS on students’ satisfaction with online learning, and rarely connect the 
assessment with students’ satisfaction directly. In particular, literature in online assessment is mostly 
theoretical, and lack of empirical testing. Our research therefore sought support for the following two 
hypotheses using empirical survey data: 

HI: Student satisfaction with online assessment is affected by their experience with the LMS, 

the ease of accessing the functionalities of the LMS, and the interaction between instructors and students. 

H2: Overall student satisfaction with online learning is affected by the extent of online 

assessment activities, experience with the LMS, the ease of accessing the functionalities of the LMS, and the 
interaction between instructors and students. 

3. METHDOLOGY 

The survey questionnaire was sent anonymously to all students who took the online or hybrid courses at a 
private Midwest university in summer 2013, as well as all instructors who taught those courses. All 
questions from the questionnaire are included in the appendix. For testing our hypotheses, data analysis is 
focused on the data collected from students. Additionally, we include summary responses from the faculty 
survey in the Conclusion section; these responses help to build the connections between the student and 
faculty perspectives. 

3.1 Sample 

The sample for the study included all 26 online or hybrid courses taught in summer 2013. Approximately a 
quarter of the students enrolled in accounting classes in the College of Business, another quarter was from the 
College of Liberal Arts and Sciences, and another quarter was from the College of Education. In total, 319 
students were enrolled in those sections, and we received responses from 9 1 students, for a response rate of 
27.7%. There were in total 22 instructors who taught the online or hybrid course sections, and we received 
responses from 17 of them, for a response rate of 77.3%. 
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3.1.1 Variable Definitions 

A majority of the students agreed or strongly agreed with the following questionnaire statements: (1) Moodle 
is straightforward and easy, (2) The organization and sequence of course was easy to navigate, (3)1 am able 
to complete class assignments in Moodle, and (4) In the majority of my courses, I interact and do things with 
content rather than read/view the content. We were reassured by the fact that our participants had good 
experiences with Moodle and relied on it to complete their class assignments, including doing things beyond 
simply viewing documents. Further, we conducted a factor analysis for all the items and found that there 
were four components and five distinct constructs. Table 1 shows that these four items loaded together on a 
single factor in Component 1, which we refer to as the “Moodle Experience” labeled (M Exp). In addition, 
the Component 1 factor structure shows that there is another distinct factor that captures the experience with 
accessing materials (refer to as M_Access). M_Access and the other constructs from our confirmatory factor 
analysis, provide us the following results shown in Table 1. 


Table 1. Confinnatory Factor Analysis - Principal Components Analysis with Varimax Rotation 


Items 

Component 1 

Component 2 

Component 3 

Component 4 

Orient M 

0.076 

0.108 

-0.007 

0.944 

Adequate Orient 

-0.013 

-0.018 

-0.069 

0.940 

View Grades 

0.429 

0.209 

0.656 

-0.126 

AccessFeedback 

0.268 

0.259 

0.824 

0.007 

Turnitin Comments 

0.172 

0.288 

0.832 

0.002 

ViewFolder 

0.813 

0.349 

0.213 

-0.014 

View Web 

0.853 

0.222 

0.251 

0.021 

Complete Assign 

0.847 

0.251 

0.189 

0.047 

Complete Turnitin 

0.749 

0.301 

0.142 

0.137 

View Panopto 

0.612 

0.418 

-0.022 

0.020 

Take Quiz 

0.584 

0.181 

0.316 

0.263 

News Forum 

0.496 

0.469 

0.356 

-0.102 

Discuss M 

0.575 

0.318 

0.492 

-0.051 

Email M 

0.427 

0.558 

0.440 

-0.051 

Chat M 

0.282 

0.694 

0.361 

-0.003 

WikiM 

0.194 

0.790 

0.298 

0.148 

Blog M 

0.282 

0.822 

0.192 

0.043 

AdobeM 

0.289 

0.724 

0.127 

0.038 

Straight Forward 

0.662 

0.220 

0.417 

0.003 

Course_Seq_Navigate 

0.751 

0.116 

0.365 

0.010 

CompleteAssign 

0.743 

0.121 

0.223 

-0.036 

Interact Do Things 

0.491 

0.296 

0.063 

-0.053 


Rotation converged in 6 iterations. Four factors extract 71% of the variance. Kaiser-Meyer-Olkin Measure of Sampling 
Adequacy = 0.914, Approx. Chi Square = 503.188. Bartlett’s Test of Sphericity df= 325, Sig. = 0.000 


In Table 1, Component 1 has six questions about access to course content and two 
communication/collaboration questions loaded together on a single factor, which is labeled as M_Access. 
The remaining five communication and collaboration items loaded together on a single factor as Component 
2, which we labeled as M_Comm. These constructs included items such as messaging, using IM/e-mail, chat, 
wiki, blogs, and Adobe Connect. The M Assess construct consists of three items shown as Component 3 in 
Table 1, and are related to conducting student assessment in Moodle. The first item measures the ease with 
which one can view grades, the second item measures the ease of accessing feedback on assignments, and the 
third question measures the ease with which one can view comments provided for assignments submitted via 
the Turnitin function of Moodle. Component 4 captures the adequacy of training provided to students, and 
we label it as “Train”. 

Table 2 shows that the “Train”, “M_Assess”, “M_Access,” and “M_Comm” variables have high 
Cronbach Alpha scores (i.e., 0.91, 0.89, 0.95, 0.96, and 0.83 respectively), which is well above the acceptable 
reliability score of 0.70 (Nunally, 1970). Hence, we conclude that the scales used are reliable. Our main 
dependent variable is a single questionnaire item measuring the overall satisfaction with Moodle. 
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Additionally, we also use M_Assess as a dependent variable to examine which factors affect online 
assessment. Since the university offered voluntary Moodle training opportunities for its faculty and students, 
we control for familiarity with it using two questions, which also loaded on a single factor (labeled as 
“Train”). Additionally, we control for students' gender, class standing, the college to which they belong, and 
their GPA. 


Table 2. Descriptive Statistics 


Variables 

Mean 

Std. Deviation 

No. Items 

Cronbach Alpha 

Overall Satisfaction 

2.20 

1.03 

1 


Train 

1.81 

0.38 

2 

0.91 

M_Assess 

3.54 

1.15 

3 

0.89 

M_Access 

3.94 

0.92 

8 

0.95 

M_Comm 

3.32 

1.03 

5 

0.96 

M_Exp 

3.90 

0.80 

4 

0.83 

Gender 

1.77 

0.42 

1 


Standing 

2.91 

0.97 

1 


GPA 

5.46 

0.87 

1 


College 

3.44 

1.85 

1 



Labels are described in the Appendix. Sample size used to calculate all the means is 91. 


Turnitin is a third-party (paid) plug-in that enhances the core Moodle system. Its digital plagiarism- 
detection utility that gives teachers the ability to grade and check papers for plagiarism. Because plagiarism 
is a growing concern in on-line education, Turnitin is an effective way for instructors to check for it 
(Sutherland- Smith & Carr, 2005). The system compares student work to past papers, articles, and books 
from internal sources as well as other university systems and provides color-coded originality scores and 
reports for use by instructors (Dahl, 2007). The system also has many additional features; for example, 
Turnitin provides digital feedback, an online grade book, and attendance lists. 

3.1.2 Models Analyzed 

The first model uses ease of assessment tools (M Assess) in Moodle as the dependent variable. An OLS 
regression analysis was performed to report the results. The “i” subscript attached to each variable refers to 
the fact that response rates were obtained from separate students. 

M Assessi = Constant + ai M_Accessi + <X2 MComrm + a3M_Expi 

+ 04 Traim + as Genden + 06 Standingi + a 7 GPAi + as Collegei + Sit ( 1 ) 

The second model uses the “Overall Satisfaction” experience of each student with using Moodle as the 
dependent variable. Since M Assess and OverallSatisfaction of students with Moodle are jointly 
determined we use two-staged least square (2SLS) analysis to test the incremental effect of M_Assess on 
Overall Satisfaction. 2SLS analysis of Overall Satisfaction as the dependent variable and M Assess as the 
instrumental variable controls for the endogenous relationship between them. 

Overall_Satisfactionit = Constant + Pi M_Assessi + P 2 M_Accessi + PsM Comirii 

+ P 4 MExpi + Ps Traim + P6 Genden + P 7 Standingi + Ps GPAi + p? Collegei + Sit (2) 


4. RESULTS 

Table 2 descriptive statistics indicate that most of our respondents were female sophomore students taking 
summer classes offered by the College of Liberal Arts and Sciences or the College of Business. The students 
typically took two or three courses over two summer sessions, and most of their courses used Moodle, which 
they valued (i.e., felt it was important) in their classes. In terms of how students used Moodle, 68.2% felt it 
was straightforward and intuitive, 75 . 9 % found it was easy to navigate, and 85.8% reflected that it enabled 
them to easily complete class assignments. Overall, only 11.3% were dissatisfied after using Moodle for 
these courses (70.8% satisfied and 18% neutral). It is worth noting that even though the majority of students’ 
comments indicated that Moodle was easy to use, only 17.6% of the students received Moodle training. 
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In comparing the subsection of 72 students whom did not receive an orientation to Moodle, the findings 
were consistent with the findings from those who did receive Moodle training. 

Table 2 shows that the mean score for “Overall Satisfaction” is 2.2 (where a score of 1 is very satisfied, 2 
is satisfied, and 3 is neutral), and the majority of students felt it was easy or very easy to view grades and 
access assignment feedback in Moodle. Nevertheless, only 43% of students felt it’s easy or very easy to use 
when the written assignments were submitted via the Turnitin functionality. The range of this response 
distribution resulted in an average score of 3.5 out of 5 for the M_Assess variable. 

The range of response rates for questions, which measured the ease of accessing (i.e., M Access) the 
various communication and collaboration functions in Moodle, varied from a low of approximately 52% for 
viewing Panopto (lecture capture) recordings to a high of 86% for completing assignments using Moodle. 
This resulted in an average score of approximately 4.0 out of 5.0 for the M Access variable. Our final 
explanatory variable is labeled M Comm, and it measured the ease of use of the common 
communication/collaboration tools in Moodle. The M_Comm, or the ease of communication measure, 
ranged from a low of approximately 17% for using the blog feature to a high of 53% for sending messages 
via e-mail. This probably reflects the student expectation that, with online courses, instructors will use 
numerous advanced access features (e.g., forums, blogs, and chats) rather than simply communicating via e- 
mails. Hence, this resulted in the lowest score of 3.3 for the M_Comm variable. The highest score of 3.9 out 
of 5.0 was for the MExp measure, which is attributable to the students’ experience with Moodle. 


Table 3. Person Correlation Coefficients 


Variable 

Train 

M Assess 

M Access 

MComm 

MExp 

Gender 

Standing 

GPA 

College 

Overall 

Satisfaction 

0.02 

-0.65*** 

-0.74*** 

-0.60*** 

-0.84*** 

-0.01 

0.16 

-0.0 

-0.16 

Train 


-0.04 

0.13 

0.11 

0.03 

0.07 

-0.06 

-0.10 

0.01 

M Assess 



0.64*** 

0 jg*** 

0.62*** 

-0.12 

-0.14 

0.03 

-0.05 

MAcess 




0.76*** 

0.82*** 

0.10 

-0.31** 

0.03 

0.14 

MComm 





0.72*** 

0.16 

-0.44** 

-0.01 

-0.16 

MExp 






0.05 

-0.20 

0.07 

0.14 

Gender 







-0.16 

0.29** 

0.20 

Standing 








-0.16 

-0.08 


***. Correlation is significant at the 0.01 level (2-tailed); **. Correlation is significant at the 0.05 level (2-tailed) 


Table 3 shows that students’ overall satisfaction with using Moodle is significant and negatively 
correlated at the p < 0.01 level with all our explanatory variables M_Assess, M_Access, M_Comm, and 
M Exp. A lower score for the overall satisfaction dependent variable refers to greater satisfaction, and a 
higher score for the explanatory variables refers to greater ease of use of using the functionality in Moodle. 
Thus, the significant negative relationship suggests that the higher the score for the explanatory variable, the 
greater the level of satisfaction with Moodle. Table 3 also shows that the student’s class standing is an 
important control variable. Our summer course data suggests that freshmen and sophomore students are 
more comfortable with the assessment and access functionalities of Moodle as compared to juniors, seniors, 
and graduate students; this is possibly explained by older students' having to “unlearn” the previous campus 
LMS. 

Table 4 shows the OLS regression analyses with M_Assess as the dependent variable. It shows that the 
M_Assess variable is significantly and positively associated with the M_Comm, and that our model 1 
explains 57% of the variance in M_Assess that is significant at p < 0.01 level. This result suggests that using 
Moodle to do online assessment activities, such as providing feedback on assignments, is significant and 
positively influenced by the extent to which instructors use the communication and collaboration tools. 
Thus, we conclude that there is significant support for HI that using Moodle for assessment activities is 
significantly and positively associated with the instructors’ familiarity with using the tools, particularly the 
advanced functionalities offered by the LMS. Further, the table also shows that gender, particularly females 
are more critical than males of learning assessments in Moodle. 
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Table 4. Regression of Assessment in Moodle (M Assess) on Explanatory Variables and Control Variables 


Variable 

Beta 

Std. Error 

t 

Sig. 

Constant 

-0.10 

1.91 

-0.05 

0.96 

M Access 

0.05 

0.35 

0.17 

0.87 

M Comm 

0.53 

0.29 

2.21 

0.04** 

M Exp 

0.31 

0.38 

1.13 

0.27 

Train 

0.06 

0.43 

0.42 

0.68 

Gender 

-0.25 

0.39 

-1.69 

0.10* 

Standing 

-0.06 

0.20 

-0.40 

0.69 

GPA 

0.07 

0.22 

0.48 

0.64 

College 

-0.01 

0.10 

-0.01 

0.99 


Adjusted R 2 = 0.57, Model’s F stat 5.31 that is significant at the 0.002 level; **. Correlation is significant at the 0.05 


level. 

Table 5 shows that the Overall Satisfaction with using the Moodle LMS is significantly and negatively 
associated with the M_Exp variable, and that our model 2 explains 60% of the variance in the level of student 
satisfaction that is significant at the p < 0.01 level. This result suggests that the higher the positive 
experience with navigating the system, completing assignments, and interaction to do things in Moodle, the 
greater the overall student satisfaction. Thus, we conclude that there is significant support for the hypothesis 
that satisfaction with using Moodle for online learning is significantly influenced by the instructors’ 
familiarity and knowledge about how best to structure the navigation of the courses in Moodle as well as the 
use of it to complete assignments. 


Table 5. 2SLS Regression of Overall Satisfaction with Moodle on Explanatory and Control Variables 


Variable 

Beta 

Std. Error 

t 

Sig. 

Constant 

5.67 

1.84 

3.08 

0.01*** 

M Assess 

-0.03 

0.23 

-0.11 

0.91 

M Access 

0.03 

0.34 

0.12 

0.91 

MComm 

0.03 

0.32 

0.09 

0.93 

M Exp 

-0.90 

0.38 

-3.30 

0.00*** 

Train 

0.05 

0.41 

0.34 

0.74 

Gender 

0.38 

0.41 

0.93 

0.37 

Standing 

0.04 

0.19 

0.30 

0.77 

GPA 

0.05 

0.22 

0.35 

0.73 

College 

-0.05 

0.10 

-0.38 

0.71 


Adjusted R 2 = 0.60, Model’s F stat 5.38 that is significant at 0.001 level. 


5. CONCLUSION 

In comparing students’ perceptions with that of the 17 faculty members, we found that faculty members felt 
the Moodle interface was straightforward (50%), easy to navigate (80%), and satisfactory to create class 
assignments (81%). Of the faculty members, 82% had been trained to use the LMS. Additionally, no one 
was dissatisfied after using Moodle during summer 2013, which was the first time that online courses were 
offered by this university. In terms of the more assessment-based and interactive/collaborative activities 
within Moodle, faculty reported less frequent use than did students of Moodle assignments (63%), Turnitin 
assignments (35%), graded discussion forums (35%), messages (35%), wikis (34%), quizzes (29%), chats 
(29%), and blogs (6%). However, 100% of instructors were 1) comfortable with Moodle, 2) able to add their 
syllabi and make their courses available, and 3) able to easily upload files and content to their sites. 
Additionally, 94% felt they were able to communicate effectively with their students using Moodle. These 
four areas were focal points for the university when providing professional development to the faculty. As a 
result, instructors were less likely to use more advanced functions of the LMS (e.g., Turnitin assignments, 
graded forum discussions, quizzes, and chats), beyond those basic functions in which they received training. 
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Hence, the faculty finding further supports our hypothesis that using Moodle for assessment activities is 
significantly and positively associated with the instructors’ familiarity with using such tools, particularly the 
advanced functionalities offered by the system. Furthermore, there is support for the hypothesis that 
satisfaction with using Moodle for online learning is significantly influenced by the instructors’ familiarity 
and knowledge about how best to structure the navigation of the courses in Moodle and use it to complete 
assignments. 
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